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Introduction 


Admission offices at colleges and universities often use SAT® scores to make decisions 
about applicants for their incoming class. Many institutions use prediction models to quantify 
a student’s potential for success using various measures, including SAT scores (NACAC, 
2016). In March 2016, the College Board introduced a redesigned SAT that better reflects 
the work that students do in college. The new test focuses on the core knowledge and skills 
that evidence shows to be critical in preparing for college and career. This is the overall 
difference between the tests and the most important one. However, in order for us to more 
clearly present our findings in this comparison study, we would like to note the following 
specific differences in format and scoring. The old SAT had three sections: critical reading, 
mathematics, and writing. The new SAT has two sections, Evidence-Based Reading and 
Writing, and Math. It also offers an optional essay. The new test reports additional scores 
(e.g., test scores, cross-test scores, and subscores). The old SAT had a total scale range of 
600-2400. The new SAT has a total scale range of 400-1600. 


Following the release of the redesigned SAT, institutions are especially interested in 
learning if and how the use of new SAT scores impacts decision-making methods, and 
studies have been undertaken to address this interest (e.g., Marini, Shaw, Young, & Walker, 
2016; Marini, Shaw, & Young, in press). The current study explores whether old and new 
SAT scores rank students within an institution in a similar way. If the rankings of students 
produced by the two different SAT scores from the old and new exam are preserved, then 
the decisions based on performance on the old exam would be essentially identical to 
decisions based on performance when the new exam is used instead. Conversely, if the 
rankings of students are markedly different, then decisions would not be consistent across 
the old and new exams. Ranking students by standardized test scores is not something that 
admission offices necessarily do. However, this study design can provide insight into 
whether or not scores on the old and new SAT place students within the same relative 
position among all students applying to an institution. This can signal whether to expect 
large, small, or essentially no changes to processes, policies, and procedures incorporating 
SAT scores on campus. 


The purpose of this study is not to validate a method of admitting students based on test 
score ranking within an institution. Rather, the aim is to examine whether the new and old 
test scores similarly rank-order applicants within an institution to better understand if any 
broad changes would be expected in the interpretation of student test scores received within 
an institution. 


Method 


To answer the question of whether the new and old SAT scores are similarly rank-ordering 
applicants within an institution, students were ranked by old and new SAT section and test 
scores within an institution from highest to lowest. Then the corresponding old and new 
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score pairs (e.g., old SAT Writing compared to new SAT Writing and Language) were 
compared to see if significant ranking differences existed. Ties in ranking could be 
computed in three different ways? (highest rank, lowest rank, or mean rank), and all three 
methods were investigated. In the highest rank method, ties are given the rank of the 
highest position that they fall in sequence, and then the ranking continues with the position 
of the next value in order. The lowest rank method is like the highest, but in reverse. Here 
ties are given the rank of the lowest position that they fall in sequence, and then the ranking 
continues with the position of the next value in order. The mean rank method finds the 
average position of the tied values and assigns that mean value as the rank for those tied 
values. The ranking then continues with the position of the next value in order. 


Sample 


The data analyzed in this study are from the pilot predictive validity study of the new SAT 
(see Shaw et al., 2016) and include 2,050 students from 15 four-year institutions (see Table 
1). Compared to the 2014 College-Bound Seniors Cohort? (the population), this study 
sample was relatively representative of African American students (13% for both), Hispanic 
students (17% sample, 18% population), and white students (46% sample, 49% population). 
However, this study sample had more Asian students (20% sample, 12% population) and 
female students (64% sample, 53% population) than the College-Bound Seniors 2014 
cohort (College Board, 2014). Please see Shaw et al. (2016) for in-depth information 
regarding sample selection and data-cleaning procedures. 


Table 1: Characteristics of Sample 
—S>SESE>>EE>E>E>>E>E>E>EEEE>>>EEEEEEEEEEEEEEEEeEE—————————————————— 


Student Characteristics (n = 2,050) % 
Gender Female 64% 
Male 36% 
Race/Ethnicity American Indian or Alaska Native <1% 
Asian, Asian American, or Pacific 20% 
Islander 
Black or African American 13% 


1 Ties can also be computed using the sequential method where all tied scores are given the same rank, and then the next 
score is ranked in sequential order with no ranks being skipped. This method was not employed due to the number of tied 
scores within a section or test within an institution. Further, the scale of the tests in the new SAT (10-40) had fewer score 
points than all sections combined. Using the sequential method for ties on data with many ties and small scales causes 
significant differences to be identified when they are actually functions of the tie method rather than actual differences 
within the data. 


? The cohort includes college-bound students in the class of 2014 who took the SAT or SAT Subject Tests™ at any time during 
high school. 
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Hispanic 17% 
White 46% 
Other 3% 
No Response <1% 
Institutional Characteristic (n = 15) % 
Control Private 33% 
Public 67% 
Admittance Rate Under 50% 40% 
50% to 75% 40% 
Over 75% 20% 
Undergraduate Enrollment Small 0% 
Medium 33% 
Large 13% 
Very Large 53% 


Measures 


Old SAT Scores. The most recent scores from the old SAT were obtained from the College 
Board for each student in this sample. The old SAT has three sections: critical reading (CR), 
mathematics (M), and writing (W). Each section has a 200-800 scale range. Ranks were 
created for each of the three sections independently, as well as for the sum of the CR and 
W sections. 


New SAT Scores. New SAT scores were obtained for each student in the study sample ina 
special administration of a pilot form of the new SAT in fall 2014. The new SAT has two 
section scores, three test scores, two cross-test scores, and seven subscores. For this 
study, we were interested in the following scores: 


* Two section scores (200-800 scale each)—Evidence-Based Reading and Writing 
(ERW) and Math (MSS). 


° Two test scores (10—40 scale each)—Reading (R) and Writing and Language (WRLA). 


Analyses and Results 


Research question: How does student ranking differ when old versus new SAT 
section and/or test scores are used to rank students within an institution? 


Before the analyses could begin, students needed to be ranked based on old SAT scores 
and new SAT scores within an institution. As described earlier, three methods for handling 
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ranking ties were employed. Therefore, a student could have three different ranks for each 
score. Students were ranked by the following scores within institutions and are listed below 
by their comparative ranking pairs. 


Old SAT Scores New SAT Scores 

Mathematics Section (SAT-M) Math Section (MSS) 

Critical Reading Section (SAT-CR) Reading Test (R) 

Writing Section (SAT-W) Writing and Language Test (WRLA) 
Critical Reading Section + Writing Section Evidence-Based Reading and Writing 
(SAT-CR+W) Section (ERW) 


Student ranks were compared within an institution and by the tie-ranking method in pairs of 
corresponding sections and/or tests on the old and new SAT. These comparisons were 
done three times, one for each method of ranking ties. Rank pairs were analyzed using the 
Wilcoxon signed-rank test, which tests for significant differences in ranks when ranks 
assigned from old and new SAT scores are compared. Tables 2—4 show the results of the 
Wilcoxon signed-rank test for each method of ranking ties. 


Table 2: Wilcoxon Signed-Rank Test Results for Ties Using the Mean Method 
aa | 


Rank of MSS Rank of ERW Rank of WRLA Rank of R 
compared to compared to rank compared to compared to 

Institution n rank of SAT-M of SAT-CR+W rank of SAT-W | rank of SAT-CR 

al 475 -0.62 -0.27 -0.55 -0.05 

2 157 -0.56 -0.52 -0.20 -0.36 

3 137 -0.13 -0.62 -0.22 -0.31 

4 146 -0.49 -0.77 -0.19 -0.53 

5 68 -0.18 -0.55 -0.21 -0.13 

6 107 -0.07 -0.21 -0.35 -0.20 

7 123 -0.23 -0.62 -0.05 -0.67 

8 113 -0.11 -0.13 -0.44 -0.07 

9 136 -0.33 -0.06 -0.06 -0.11 

10 83 -0.13 -0.49 -0.43 -0.43 

11 224 -0.26 -0.40 -0.23 -0.13 

12 103 -0.44 -0.55 -0.49 -0.44 

13 182 -0.25 -0.95 -0.51 -1.02 
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14 
15 


96 
200 


-0.48 
-0.51 


-0.31 
-0.71 


-0.19 
-0.05 


-0.30 
-0.36 


Table 3: Wilcoxon Signed-Rank Test Results for Ties Using the Low Method 
SSS SS ee 


Institution 


oO AN DO Oo FP W DN KF 


15 


*p<0.05 


n 
175 
157 
137 
146 

68 
107 
123 
113 
136 

83 
224 
103 
182 

96 
200 


Rank of MSS 
compared to 


rank of SAT-M 


-0.84 
-0.47 
-0.21 
-0.32 
-0.24 
-0.15 
-0.21 
-0.11 
-0.29 
-0.29 
-0.14 
-0.40 
-0.39 
-0.57 
-0.38 


Rank of ERW 


compared to rank 
of SAT-CR+W 


-0.81 
-1.37 
-1.10 
-1.08 
-0.81 
-0.81 
-1.38 
-0.78 
-0.59 
-0.98 
-0.94 
-0.92 
-1.37 
-0.96 
-1.65 


Rank of WRLA 
compared to 
rank of SAT-W 


-1.65 
-1.40 
-1.09 
-1.10 
-0.82 
-1.34 
-1.13 
-1.38 
-0.98 
-1.19 
-1.23 
-0.88 
-1.29 
-0.80 
-1.49 


Rank of R 


compared to 
rank of SAT-CR 


-1.32 
-2.01* 
-0.77 
-1.15 
-0.59 
-0.66 
-1.67 
-0.93 
-0.86 
-0.80 
-1.13 
-1.42 
-1.81 
-1.00 
-1.57 


Table 4: Wilcoxon Signed-Rank Test Results for Ties Using the High Method 
| 


Institution 


1 
2 
3 
4 
5 
6 


n 
175 
157 
137 
146 

68 
107 


Rank of MSS 
compared to 


rank of SAT-M 


-0.37 
-0.62 
-0.02 
-0.66 
-0.19 
-0.05 


Rank of ERW 


compared to rank 
of SAT-CR+W 


-0.25 
-0.39 
-0.10 
-0.37 
-0.28 
-0.36 


Rank of WRLA 
compared to 
rank of SAT-W 


-0.76 
-1.06 
-0.70 
-0.77 
-0.42 
-0.69 


Rank of R 


compared to 
rank of SAT-CR 


-1.22 
-1.31 
-0.21 
-0.16 
-0.34 
-0.38 
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7 123 -0.28 -0.10 -1.06 -0.46 
8 113 -0.12 -0.43 -0.53 -0.76 
9 136 -0.34 -0.53 -1.03 -0.67 
10 83 -0.01 -0.04 -0.44 -0.03 
11 224 -0.35 -0.16 -0.68 -0.83 
12 103 -0.48 -0.20 -0.01 -0.56 
13 182 -0.09 -0.52 -0.26 -0.16 
14 96 -0.37 -0.24 -0.50 -0.53 
15 200 -0.57 -0.24 -1.36 -0.78 


Almost all results of the Wilcoxon signed-rank test showed that the differences in ranks 
between old and new SAT section/test score pairs were not significant. Out of 180 
comparisons, only one significant difference in rank was found. This significant difference 
was found within Institution 2 (of 15) when the rank of the old SAT critical reading section 
was compared with the rank of the new SAT Reading test. In light of the fact that there was 
no adjustment to our overall alpha level (each test used an alpha of .05) finding, at least one 
significant result by chance was a virtual certainty. 


Discussion and Conclusion 


The findings from this study suggest that if students were ranked within an institution using 
old and new SAT scores, their relative position within an institution (among applicants or 
enrolled students) would not change in any meaningful way. Of the 180 comparisons of 
student rank by old and new SAT scores, only one comparison indicated a statistical 
difference in student ranking between old and new scores (at one institution, comparing SAT 
critical reading section scores to SAT Reading test scores). The overwhelming 
preponderance of evidence indicates that admission professionals can continue to rely on 
the new SAT for making admission decisions and can expect few if any major disruptions or 
changes in the use of scores with regard to academically rank-ordering applicants. These 
results are in keeping with other pilot research studies examining new SAT scores. Previous 
results indicate that the new test scores maintain their strong predictive validity with regard 
to the first-year grade point average (Shaw et al., 2016) and that the use of old SAT scores, 
new SAT scores, and concorded SAT scores can all serve institutions well during the 
transition period from using old SAT scores to using new SAT scores (Marini et al., 2016; 
Marini et al., in press). 
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